Learning Sequential Patterns for Lipreading

This paper presents a machine learning approach to Lip Reading and proposes a novel learning technique called sequential pattern boosting that allows us to efficiently search and combine temporal patterns to form strong spatio-temporal classifiers. Attempts at automatic lip reading need to address the demanding challenge that the problem is inherently temporal in nature. It is crucial to model and use spatio-temporal information. To achieve this, we use sequential patterns, an ordered sequence of feature subsets. Sequential patterns then form weak classifiers that are combined together into a strong spatio-temporal classifier by means of boosting. A boosted classifier consists of a linear combination of a number (S) of selected weak classifiers, and take the form of: H(I) = ∑i=1 αihi(I). The weak classifiers hi are selected iteratively based on weights formed during training. In order to determine the optimal weak classifier at each Boosting iteration, the common approach is to exhaustively search the entire set of candidate weak classifiers. However, when dealing with sequential patterns, the number of weak classifiers becomes too large. Specifically, given D features, sequential patterns up to length N and maximum of K items, the number of weak classifiers is the binomial coefficient polynomial: (D K )N . In the experiments performed here (D = 900,K = 3,N = 7), the total number of weak classifiers is 5x1058. To address this, we propose a novel Boosting algorithm called Sequential Pattern Boosting (SPBoost). Firstly, we define sequential patterns as follows: